Improving the Throughput of Remote Storage Access through Pipelining

Data intensive applications constitute a large and increasing share of Grid computing. However there are relatively few results on how to improve the efficiency of the basic data transfer mechanisms used to move large data set in and out of clusters. In this paper we describe a simple and general technique to improve the throughput of data transfer protocols and we demonstrate it on the SDSC Storage Resource Broker (SRB), a remote storage access middleware for supercomputer applications. We achieve a maximum performance improvement of 43%/52% for remote reads/writes larger than 1MB with a few changes to the original SRB protocol. The protocol was restructured by introducing a notion of pipelining that enables the overlapping of the various stages of the data processing, such as network transfer and disk access. We present a detailed analysis of the pipelined SRB implementation and the pipeline cost model we used to drive our design.